Methods and apparatuses for use in switching between streaming video bitstreams

ABSTRACT

Improved methods and apparatuses are provided for switching of streaming data bitstreams, such as, for example, used in video streaming and other related applications. Some desired functionalities provided herein include random access, fast forward and fast backward, error-resilience and bandwidth adaptation. The improved methods and apparatuses can be configured to increase coding efficiency of and/or reduce the amount of data needed to encode a switching bitstream.

RELATED PATENT APPLICATIONS

[0001] This U.S. Non-provisional Application for Letters Patent furtherclaims the benefit of priority from, and hereby incorporates byreference the entire disclosure of, co-pending U.S. ProvisionalApplication for Letters Patent Serial No. 60/355,071, filed Feb. 8,2002.

[0002] Furthermore, this U.S. Non-provisional Application for LettersPatent is related to a co-pending application Ser. No. ______(Attorney's Docket Number MS1-1218US), filed Jun. 27, 2002, and titled“Seamless Switching Of Scalable Video Bitstreams”.

TECHNICAL FIELD

[0003] This invention relates to data bitstreams, and more particularlyto methods and apparatuses for switching between different streamingbitstreams.

BACKGROUND

[0004] With steady growth of access bandwidth, more and more Internetapplications start to use streaming audio and video contents. Since thecurrent Internet is inherently a heterogeneous and dynamical best-effortnetwork, channel bandwidth usually fluctuates in a wide range from bitrate below 64 kbps to well above 1 Mbps. This brings great challenges tovideo coding and streaming technologies in providing a smooth playbackexperience and best available video quality to the users. To deal withthe network bandwidth variations, two main approaches, namely, switchingamong multiple non-scalable bitstreams and streaming with a singlescalable bitstream, have been extensively investigated in recent years.

[0005] In the first approach, a video sequence is compressed intoseveral non-scalable bitstreams at different bit rates. Some specialframes, known as key frames, are either compressed without prediction orcoded with an extra switching bitstream. Key frames provide accesspoints to switch among these bitstreams to fit in the availablebandwidth. One advantage of this method is the high coding efficiencywith non-scalable bitstreams. However, due to limitation in both thenumber of bitstreams and switching points, this method only providescoarse and sluggish capability in adapting to channel bandwidthvariations.

[0006] In the second approach, a video sequence is compressed into asingle scalable bitstream, which can be truncated flexibly to adapt tobandwidth variations. Among numerous scalable coding techniques, MPEG-4Fine Granularity Scalable (FGS) coding has become prominent due to itsfine-grain scalability. Since the enhancement bitstream can be truncatedarbitrarily in any frame, FGS provides a remarkable capability inreadily and precisely adapting to channel bandwidth variations. However,low coding efficiency is the vital disadvantage that prevents FGS frombeing widely deployed in video streaming applications. Progressive FineGranularity Scalable (PFGS) coding scheme is a significant improvementover FGS by introducing two prediction loops with different qualityreferences. On the other hand, since only one high quality reference isused in enhancement layer coding, most coding efficiency gain appearswithin a certain bit rate range around the high quality reference.Generally, with today's technologies, there is still a coding efficiencyloss compared with the non-scalable case at fixed bit rates.

[0007] Nevertheless, bandwidth fluctuations remain a problem forstreaming video in the current Internet. Conventional streaming videosystems typically try to address this problem by switching betweendifferent video bitstreams with different bit-rates, for example, asdescribed above. However, in these and other existing video codingschemes, the switching points are restricted only to key frames (e.g.,typically I-frames) to avoid drifting problems. Such key frames areusually encoded far apart from each other to preserve high codingefficiency, so bitstream switching can only take place periodically.This greatly reduces the adaptation capability of existing streamingsystems. Consequently, a viewer may experience frequent pausing andre-buffering when watching a streaming video.

[0008] Hence, there is a need for improved method and apparatuses foruse in switching streaming bitstreams.

SUMMARY

[0009] Improved methods and apparatuses are provided for switching ofstreaming data bitstreams, such as, for example, used in video streamingand other related applications. Some desired functionalities providedherein include random access, fast forward and fast backward,error-resilience and bandwidth adaptation. The improved methods andapparatuses can be configured to increase coding efficiency of and/orreduce the amount of data needed to encode a switching bitstream.

[0010] In accordance with certain exemplary implementations of thepresent invention, an encoding method is provided. The method includesencoding data into a first bitstream using a first quantizationparameter and encoding the data into a second bitstream using a secondquantization parameter that is different from the first quantizationparameter. The method also includes generating an encoded switchingbitstream associated with the first and second bitstreams using thefirst quantization parameter to support up-switching between the firstand second bitstreams and using the second quantization parameter tosupport down-switching between the first and second bitstreams.

[0011] An exemplary apparatus includes a first bitstream encoderconfigured to encode data into an encoded first bitstream using a firstquantization parameter and a second bitstream encoder configured toencode the data into an encoded second bitstream using a secondquantization parameter that is different from the first quantizationparameter. The apparatus also includes a switching bitstream encoderoperatively coupled to the first bitstream encoder and the secondbitstream encoder and configured to output an encoded switchingbitstream that supports up-switching and down-switching between thefirst encoded bitstream and the second encoded bitstream based oninformation processed using the first and second quantizationparameters.

[0012] An exemplary decoding method includes receiving at least oneencoded bitstream, such as, a first bitstream that was generated using afirst quantization parameter and/or a second bitstream that wasgenerated using a second quantization parameter that is different fromthe first quantization parameter. The received encoded bitstream isdecoded. The decoding method further includes receiving an encodedswitching bitstream associated with the first and second bitstreams thatwas generated using the first quantization parameter to supportup-switching between the first and second bitstreams and using thesecond quantization parameter to support down-switching between thefirst and second bitstreams. The method also includes decoding thereceived encoded switching bitstream using the first and secondquantization parameters.

[0013] Another exemplary apparatus includes a first decoder configuredto decode a first encoded bitstream into a decoded first bitstream usinga first quantization parameter and a second decoder configured to decodea second bitstream into a decoded second bitstream using a secondquantization parameter that is different from the first quantizationparameter. The apparatus also includes a switching bitstream decoderthat is operatively coupled to the first decoder and the second decoderand configured to output a decoded switching bitstream that supportsup-switching and down-switching between the first decoded bitstream andthe second decoded bitstream based on information processed using thefirst and second quantization parameters.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings. The same numbersare used throughout the figures to reference like components and/orfeatures.

[0015]FIG. 1 is a block diagram depicting an exemplary computingenvironment that is suitable for use with certain implementations of thepresent invention.

[0016]FIG. 2 is a diagram illustratively depicting switching betweenbitstreams, in accordance with certain exemplary implementations of thepresent invention.

[0017]FIG. 3 is a block diagram depicting a conventional decoder.

[0018]FIG. 4 is a block diagram depicting a conventional encoder.

[0019]FIG. 5 is block diagram depicting an improved decoder, inaccordance with certain exemplary implementations of the presentinvention.

[0020]FIG. 6 is block diagram depicting an improved encoder, inaccordance with certain exemplary implementations of the presentinvention.

[0021]FIG. 7 is block diagram depicting an improved decoder, inaccordance with certain further exemplary implementations of the presentinvention.

[0022]FIG. 8 is block diagram depicting an improved encoder, inaccordance with certain further exemplary implementations of the presentinvention.

[0023]FIG. 9 is block diagram depicting an improved decoder, inaccordance with still other exemplary implementations of the presentinvention.

[0024]FIG. 10 is block diagram depicting an improved decoder, inaccordance with still other exemplary implementations of the presentinvention.

[0025]FIG. 11 is block diagram depicting an improved encoder, inaccordance with still other exemplary implementations of the presentinvention.

DETAILED DESCRIPTION

[0026] Ragip Kurceren and Marta Karczewicz, in a document titled“Improved SP-frame Encoding”, VCEG-M-73, ITU-T Video Coding ExpertsGroup Meeting, Austin, Tex., 02-04 April 2001 (hereinafter simplyreferred to as Kurceren et al.), proposed a switching scheme that allowsseamless switching between bitstreams with different bit-rate. Itintroduced a special frame called an SP picture that serves as aswitching point in a video sequence.

[0027] A similar representative switching process 200 is depicted in theillustrative diagram in FIG. 2. Here, switching is shown as occurringfrom bitstream 1 to bitstream 2 using SP pictures.

[0028] The streaming system usually either transmits bitstream 1 orbitstream 2, for example, depending on the current channel bandwidth.However, when the channel bandwidth changes, the transmitted bitstreamcan be switched to a bit-rate that matches the current channelcondition, for example, to improve the video quality if bandwidthincreases and to maintain smooth playback if bandwidth drops.

[0029] When switching from bitstream 1 to bitstream 2, the streamingsystem does not need to wait for a key frame to start the switchingprocess. Instead, it can switch at the SP frames. At SP frames, thestreaming system sends a switching bitstream S12, and the decoderdecodes the switching bitstream using the same techniques withoutknowing whether it is S1, S2 or S12. Thus, the bitstream switching istransparent to the decoder. The decoded frame will be exactly the sameas the reference frame for the next frame prediction in bitstream 2. Assuch, there should not be any drifting problems.

[0030] An exemplary conventional decoder 300 and encoder 400 aredepicted in FIG. 3 and FIG. 4, respectively. A more detailed descriptionof the scheme can be found in Kurceren et al. There are some potentialissues with the scheme in Kurceren et al.

[0031] For example, in real streaming applications, it is usuallydesirable to be able to switch down from a high bit-rate bitstream to alow bit-rate one very quickly. This is a desirable feature, for example,for TCP-friendly protocols currently used in many existing streamingsystems. On the other hand, switching up from a low bit-rate videobitstream to a high bit-rate does not usually have to be done as quicklyas switching down. This is again a feature of the TCP-friendlyprotocols, for example.

[0032] Therefore, it would be useful to support more rapid and frequentdown-switching. Indeed, as mentioned, the very reason for down-switchingis often related to reduced/reducing channel bandwidth capabilities. Thesize of the down-switching bitstream may often be much smaller than thatof the up-switching one. Since the high bit-rate bitstream typicallycontains most of the information of a low bit-rate one, in theory, oneshould be able to configure the scheme to make the size of switchingbitstream sufficiently small.

[0033] However, the scheme in Kurceren et al. only allows the same Qsfor both the down-switching bitstream and up-switching bitstream (see,e.g., FIG. 4), and the Qs is included in the prediction andreconstruction loop. The introduction of quantization Qs in theprediction and reconstruction loop will inevitably degrade the codingefficiency of the original bitstreams without SP frames. If one sets Qstoo small, high coding efficiency for both bitstreams 1 and 2 can beachieved. However, the difference for down-switching is also fine-grainquantized and it would result a very large down-switching bitstream.Conversely, if one sets Qs too large, although obtaining a very compactswitching bitstream, the coding efficiency of bitstreams 1 and 2 will beseverely degraded, which is not desired either. It appears that thisdemonstrative contradiction can not be solved by the techniques proposedin Kurceren et al., which make a compromise between coding efficiencyand the size of the switching bitstream.

[0034] Furthermore, there are many quantization and dequantizationprocesses in the signal flow in the encoder proposed in Kurceren et al.(see, e.g., FIG. 4). This tends to further degrade the coding efficiencyof bitstreams 1 and 2. There is also a mismatch between the predictionreference and reconstruction reference in Kurceren et al. that maycontribute to the coding efficiency degradation of bitstreams 1 and 2.

[0035] In order to address these and other issues/problems improvedmethods and apparatuses are provided herein that allow different Qs forswitching up and switching down. The block diagrams depicted in FIG. 5and FIG. 6 illustrate an improved decoder and encoder, respectively, inaccordance with certain implementations of the present invention.

[0036] In accordance with certain aspects of the present invention, theproposed techniques solve the contradiction existing in the schemeproposed in Kurceren et al. so that the down-switching bitstream can beencoded to have significantly reduced, if not minimal, size while thecoding efficiency of bitstreams 1 and 2 is also well preserved.

[0037] In accordance with certain other aspects of the presentinvention, the switching points for up-switching and down-switching canbe decoupled. This means that one can encode more switching down pointsthan switching-up points, for example, to suit the TCP-friendlyprotocols, etc. Moreover, such decoupling allows for further improvedcoding efficiency of the bitstream that the system is switched from, forexample, by individually setting the Qs in the reconstruction loop to anappropriately small value.

[0038] In accordance with certain other aspects of the presentinvention, the improved methods and apparatuses can be furthersimplified and additional quantization and dequantization processes canbe readily removed. For example, FIG. 7 and FIG. 8 illustrate anexemplary decoder and encoder, respectively, that support both highcoding efficiency for the normal bitstreams and a compact size for theswitching bitstream.

[0039] Exemplary Operational Environments:

[0040] Turning to the drawings, wherein like reference numerals refer tolike elements, the invention is illustrated as being implemented in asuitable computing environment. Although not required, the inventionwill be described in the general context of computer-executableinstructions, such as program modules, being executed by a personalcomputer.

[0041] Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types. Those skilled in the art willappreciate that the invention may be practiced with other computersystem configurations, including hand-held devices, multi-processorsystems, microprocessor based or X programmable consumer electronics,network PCs, minicomputers, mainframe computers, portable communicationdevices, and the like.

[0042] The invention may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

[0043]FIG. 1 illustrates an example of a suitable computing environment120 on which the subsequently described systems, apparatuses and methodsmay be implemented. Exemplary computing environment 120 is only oneexample of a suitable computing environment and is not intended tosuggest any limitation as to the scope of use or functionality of theimproved methods and systems described herein. Neither should computingenvironment 120 be interpreted as having any dependency or requirementrelating to any one or combination of components illustrated incomputing environment 120.

[0044] The improved methods and systems herein are operational withnumerous other general purpose or special purpose computing systemenvironments or configurations. Examples of well known computingsystems, environments, and/or configurations that may be suitableinclude, but are not limited to, personal computers, server computers,thin clients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputers, mainframe computers,distributed computing environments that include any of the above systemsor devices, and the like.

[0045] As shown in FIG. 1, computing environment 120 includes ageneral-purpose computing device in the form of a computer 130. Thecomponents of computer 130 may include one or more processors orprocessing units 132, a system memory 134, and a bus 136 that couplesvarious system components including system memory 134 to processor 132.

[0046] Bus 136 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus also known as Mezzaninebus.

[0047] Computer 130 typically includes a variety of computer readablemedia. Such media may be any available media that is accessible bycomputer 130, and it includes both volatile and non-volatile media,removable and non-removable media.

[0048] In FIG. 1, system memory 134 includes computer readable media inthe form of volatile memory, such as random access memory (RAM) 140,and/or non-volatile memory, such as read only memory (ROM) 138. A basicinput/output system (BIOS) 142, containing the basic routines that helpto transfer information between elements within computer 130, such asduring start-up, is stored in ROM 138. RAM 140 typically contains dataand/or program modules that are immediately accessible to and/orpresently being operated on by processor 132.

[0049] Computer 130 may further include other removable/non-removable,volatile/non-volatile computer storage media. For example, FIG. 1illustrates a hard disk drive 144 for reading from and writing to anon-removable, non-volatile magnetic media (not shown and typicallycalled a “hard drive”), a magnetic disk drive 146 for reading from andwriting to a removable, non-volatile magnetic disk 148 (e.g., a “floppydisk”), and an optical disk drive 150 for reading from or writing to aremovable, non-volatile optical disk 152 such as a CD-ROM/R/RW,DVD-ROM/R/RW/+R/RAM or other optical media. Hard disk drive 144,magnetic disk drive 146 and optical disk drive 150 are each connected tobus 136 by one or more interfaces 154.

[0050] The drives and associated computer-readable media providenonvolatile storage of computer readable instructions, data structures,program modules, and other data for computer 130. Although the exemplaryenvironment described herein employs a hard disk, a removable magneticdisk 148 and a removable optical disk 152, it should be appreciated bythose skilled in the art that other types of computer readable mediawhich can store data that is accessible by a computer, such as magneticcassettes, flash memory cards, digital video disks, random accessmemories (RAMs), read only memories (ROM), and the like, may also beused in the exemplary operating environment.

[0051] A number of program modules may be stored on the hard disk,magnetic disk 148, optical disk 152, ROM 138, or RAM 140, including,e.g., an operating system 158, one or more application programs 160,other program modules 162, and program data 164.

[0052] The improved methods and systems described herein may beimplemented within operating system 158, one or more applicationprograms 160, other program modules 162, and/or program data 164.

[0053] A user may provide commands and information into computer 130through input devices such as keyboard 166 and pointing device 168 (suchas a “mouse”). Other input devices (not shown) may include a microphone,joystick, game pad, satellite dish, serial port, scanner, camera, etc.These and other input devices are connected to the processing unit 132through a user input interface 170 that is coupled to bus 136, but maybe connected by other interface and bus structures, such as a parallelport, game port, or a universal serial bus (USB).

[0054] A monitor 172 or other type of display device is also connectedto bus 136 via an interface, such as a video adapter 174. In addition tomonitor 172, personal computers typically include other peripheraloutput devices (not shown), such as speakers and printers, which may beconnected through output peripheral interface 175.

[0055] Computer 130 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer182. Remote computer 182 may include many or all of the elements andfeatures described herein relative to computer 130.

[0056] Logical connections shown in FIG. 1 are a local area network(LAN) 177 and a general wide area network (WAN) 179. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets, and the Internet.

[0057] When used in a LAN networking environment, computer 130 isconnected to LAN 177 via network interface or adapter 186. When used ina WAN networking environment, the computer typically includes a modem178 or other means for establishing communications over WAN 179. Modem178, which may be internal or external, may be connected to system bus136 via the user input interface 170 or other appropriate mechanism.

[0058] Depicted in FIG. 1, is a specific implementation of a WAN via theInternet. Here, computer 130 employs modem 178 to establishcommunications with at least one remote computer 182 via the Internet180.

[0059] In a networked environment, program modules depicted relative tocomputer 130, or portions thereof, may be stored in a remote memorystorage device. Thus, e.g., as depicted in FIG. 1, remote applicationprograms 189 may reside on a memory device of remote computer 182. Itwill be appreciated that the network connections shown and described areexemplary and other means of establishing a communications link betweenthe computers may be used.

[0060] Exemplary Switching Schemes:

[0061] In this section, exemplary encoder and the decoder methods andapparatuses are described in more detail with reference to FIGS. 5-8.For comparison, additional description of the architecture proposed byKurceren et al is also provided.

[0062] The modules and notations used in FIGS. 3-8 are defined asfollows:

[0063] DCT: Discrete cosine transform.

[0064] IDCT: Inverse discrete cosine transform.

[0065] Entropy Encoding: Entropy encoding of quantized coefficients. Itcould be arithmetic coding or variable length coding.

[0066] Entropy Decoding: Entropy decoding of quantized coefficients. Itcould be arithmetic decoding or variable length decoding that matchesthe corresponding modules in the encoder.

[0067] Q: Quantization.

[0068] Q⁻¹: Inverse Quantization or dequantization.

[0069] MC: Motion compensation module, where a predicted frame is formedaccording to the motion vectors and the reference in the frame buffer.

[0070] ME: Motion estimation module, where the motion vectors aresearched to best predict the current frame.

[0071] Loop Filter: A smoothing filter in the motion compensation loopto reduce the blocking artifacts.

[0072] FrameBuffer0: A frame buffer that holds the reference frame fornext frame encoding/decoding.

[0073] P Picture: A frame encoded using traditional motion compensatedpredictive coding.

[0074] SP Picture: A frame encoded as a switching frame using theproposed motion compensated predictive coding.

[0075] Switching Bitstream: The bitstream transmitted to make seamlesstransition from one bitstream to another.

[0076] There are some basic assumptions on the quantization anddequantization:

[0077] If L₁=Q(K₁), Q(Q⁻¹(K₁))=L₁;

[0078] If L₁=Q(K₁) and L₂=Q(K₂), Q(Q⁻¹(L₁)+Q⁻¹(L₂))=L₁+L₂;

[0079] If L₁=Q(K₁), Q(Q⁻¹(L₁)+K₂))=L₁+Q(K₂);

[0080] The following descriptions work for the inter-macroblocks in aframe. For intra-macroblocks, a simple “copy” operation can be used.

[0081] Reference is now made to the conventional decoding processillustrated, for example, in FIG. 3. Here, the decoding of SP frame S1or S2 in a normal bitstream is shown. Using S1 as an example, afterentropy decoding of the bitstream S1, the levels of the prediction errorcoefficients, L_(err1), and motion vectors, are generated for themacroblock. Levels L_(err1) are dequantized using dequantizer QP₁ ⁻¹:

[0082] K_(serr1)=QP₁ ⁻¹(L_(err1)).

[0083] After motion compensation, perform forward DCT transform for thepredicted macroblock and obtain K_(pred1), the reconstructedcoefficients K_(rec1) are obtained by:

[0084] K_(rec1)=K_(pred1)+K_(serr1).

[0085] The reconstructed coefficients K_(rec1) are quantized by Qs toobtain reconstructed levels L_(rec1),

[0086] L_(rec1)=Qs(K_(rec1)).

[0087] The levels L_(rec1) are dequantized using Qs⁻¹ and the inverseDCT transform is performed to obtain the reconstructed image. Thereconstructed image will go through a loop filter to smooth certainblocky artifacts and output to the display and to the frame buffer forthe next frame decoding.

[0088] When decoding of switching bitstream S12, e.g., when switchingfrom bitstream 1 to bitstream 2, the decoding process is the same as thedecoding of S1, except that the input is bitstream S12, QP₁ ⁻¹ isreplaced by Qs⁻¹, K_(serr1) is replaced by K_(serr12), L_(rec1) isreplaced by L_(rec2), K_(rec1) is replaced by K_(rec12), and L_(err1) isreplaced by L_(err12).

[0089] The resultant picture is the same as that decoded from S2. Thus adrifting-free switching from bitstream 1 to bitstream 2 is achieved. Qsis encoded in the S12 bitstream.

[0090] Reference is now made to FIG. 4 and the exemplary conventionalencoding process that is illustrated for encoding of SP frame S1 or S2in a normal bitstream. Here, S1 is used as an example.

[0091] DCT transform to the macroblock of the original video isperformed, and the obtained coefficients as K_(ong1) denoted. Aftermotion compensation, a DCT transform is performed to the predictedmacroblock, and the obtained coefficients as K_(pred1) denoted. The nextstep is to quantize K_(pred1) using Qs and obtain levels L_(pred1).

[0092] L_(pred1)=Qs(K_(pred1)).

[0093] Then dequantize L_(pred1) using dequantizer Qs⁻¹,K_(spred1)=Qs⁻¹(L_(pred1)) and subtract K_(spred1) from K_(orig1) toobtain error coefficients K_(err1),

[0094] K_(err1)=K_(orig1)−K_(pred1).

[0095] Then quantize K_(err1) using QP₁ and obtain error levelsL_(err1),

[0096] L_(err1)=QP₁(K_(err1)).

[0097] Next, perform entropy encoding on L_(err1) and obtain bitstreamS1. Using the S1 decoder described above, for example, reconstructlevels L_(rec1) and the reference for the next frame encoding. Note thatin this example there is a quantizer Qs and a dequantizer Qs⁻¹ in thereconstruction loop.

[0098] Notice that there is a mismatch between prediction reference andreconstruction reference in this scheme. The encoding of switchingbitstream S12 (switching from bitstream 1 to bitstream 2). The encodingof S12 is based on the encoding of S1 and S2. L_(pred1) is subtracted bythe S1 encoder from the reconstructed level L_(rec2) in S2 encoder.

[0099] L_(err12)=L_(rec2)−L_(pred1).

[0100] Entropy encoding is performed with L_(err12) and bitstream S12.

[0101] An improved decoding process 500 of FIG. 5, in accordance withcertain exemplary implementations of the present invention will now bedescried in greater detail.

[0102] To describe the decoding of SP frame S1 or S2 in a normalbitstream, S1 is used as an example. After entropy decoding of thebitstream S1, the levels of the prediction error coefficients, L_(err1),and motion vectors, are generated for the macroblock. Levels L_(err1)are dequantized using quantizer QP₁ ⁻¹,

[0103] K_(serr1)=QP₁ ⁻¹(L_(err1)).

[0104] The error coefficients K_(serr1) are quantized using quantizerQs=Qs₁ and obtain levels,

[0105] L_(serr1)=Qs(K_(serr1)).

[0106] After motion compensation, forward DCT transform is performed forthe predicted macroblock and obtain K_(pred1). K_(pred1) is thenquantized by Qs₁,

[0107] L_(pred1)=Qs₁(K_(pred1)).

[0108] L_(pred1) is then dequantized by Qs₁ ⁻¹,

[0109] K_(spred1)=Qs₁ ⁻¹(L_(pred1)).

[0110] The dequantized coefficients K_(spred1) are further quantized byquantizer Qs=Qs₁ and obtain levels,

[0111] L_(spred1)=Qs(K_(spred1)).

[0112] The reconstructed levels L_(rec1) are obtained by,

[0113] L_(rec1)=L_(spred1)+L_(serr1).

[0114] The levels L_(rec1) are dequantized using Qs⁻¹=Qs₁ ⁻¹ and theinverse DCT transform is performed to obtain the reconstructed image.The reconstructed image will go through a loop filter to smooth certainblocky artifacts and output to the display and to the frame buffer fornext frame decoding.

[0115] The decoding of switching bitstream S12, for example, whenswitching from bitstream 1 to bitstream 2, follows a similar decodingprocess similar except that the input is bitstream S12, QP₁ ⁻¹ isreplaced by Qs₂ ⁻¹, Qs is replaced by Qs₂, Qs⁻¹ is replaced by Qs₂ ⁻¹,L_(rec1) is replaced by L_(rec2), L_(err1) is replaced by L_(err12),K_(serr1) is replaced by K_(serr12), and L_(spred1) is replaced byL_(spred12).

[0116] Note that the information on Qs₁ and Qs₂ is encoded in bitstreamS12.

[0117] The resultant picture is the same as that decoded from S2. Thus adrifting-free switching from bitstream 1 to bitstream 2 is achieved.

[0118] An improved encoding process 600 of FIG. 6, in accordance withcertain exemplary implementations of the present invention will now bedescried in greater detail.

[0119] To describe the encoding of SP frame S1 or S2 in a normalbitstream, S1 is used as an example. Here, for example, DCT transform isperformed to the macroblock of the original video, and the obtainedcoefficients as K_(orig1) denoted.

[0120] After motion compensation, DCT transform is performed to thepredicted macroblock, and the obtained coefficients denoted asK_(pred1). Then K_(pred1) is quantized using Qs₁ and levels L_(pred1)obtained,

[0121] L_(pred1)=Qs₁(K_(pred1)).

[0122] Then, the next step is to dequantize L_(pred1) using dequantizerQs₁ ⁻¹,

[0123] K_(spred1)=Qs₁ ⁻¹(L_(pred1)).

[0124] Then subtract K_(spred1) from K_(orig1) and obtain errorcoefficients K_(err1),

[0125] K_(err1)=K_(orig1)−K_(pred1).

[0126] Next, quantize K_(err1) using QP₁ and obtain error levelsL_(err1),

[0127] L_(err1)=QP₁(K_(err1)).

[0128] Then, perform entropy encoding on L_(err1) and obtain bitstreamS1.

[0129] Using the S1 decoder described above, for example, reconstructlevels L_(rec1) and the reference for the next frame encoding.

[0130] Note that here there is a quantizer Qs₁ and a dequantizer Qs₁ ⁻¹in the reconstruction loop.

[0131] The encoding of switching bitstream S12, for example, whenswitching from bitstream 1 to bitstream 2, is based on the encoding ofS1 and S2.

[0132] Here, the process involves quantizing prediction coefficientsK_(spred1) in the S1 encoder using quantizer Qs₂.

[0133] L_(spred12)=Qs₂(K_(spred1)).

[0134] Then subtract L_(spred12) from the reconstructed level L_(rec2)in S2 encoder.

[0135] L_(err12)=L_(rec2)−L_(spred12).

[0136] Next, perform entropy encoding of L_(err12) and generatebitstream S12.

[0137] An improved decoding process 700 of FIG. 7, in accordance withcertain further exemplary implementations of the present invention willnow be descried in greater detail.

[0138] To describe the decoding of SP frame S1 or S2 in a normalbitstream, S1 is used as an example.

[0139] Here, after entropy decoding of the bitstream S1, the levels ofthe prediction error coefficients, L_(err1), and motion vectors, aregenerated for the macroblock. Levels L_(err1) are dequantized usingquantizer QP₁ ⁻¹:

[0140] K_(serr1)=QP₁ ⁻¹(L_(err1)).

[0141] After motion compensation, perform forward DCT transform for theX predicted macroblock and obtain K_(pred1), the reconstructedcoefficients K_(rec1) are obtained by:

[0142] K_(rec1)=K_(pred1)+K_(serr1).

[0143] The reconstructed coefficients K_(rec1) are quantized by Qs₁ toobtain reconstructed levels L_(rec1),

[0144] L_(rec1)=Qs₁(K_(rec1)).

[0145] The levels L_(rec1) are dequantized using Qs₁ ⁻¹ and the inverseDCT transform is performed to obtain the reconstructed image. Thereconstructed image will go through a loop filter to smooth certainblocky artifacts and output to the display and to the frame buffer fornext frame decoding.

[0146] The decoding of switching bitstream S12, for example, whenswitching from bitstream 1 to bitstream 2, follows a similar decodingprocess similar except that the input is bitstream S12, QP₁ ⁻¹ isreplaced by Qs₂ ⁻¹, Qs₁ is replaced by Qs₂, Qs₁ ⁻¹ is replaced by Qs₂⁻¹, K_(serr1) is replaced by K_(serr12), K_(rec1) is replaced byK_(rec12), L_(rec1) is replaced by L_(rec2), and L_(err1) is replaced byL_(err12).

[0147] The resultant picture is the same as that decoded from S2. Thus,a drifting-free switching from bitstream 1 to bitstream 2 is achieved.

[0148] An improved encoding process 800 of FIG. 8, in accordance withcertain further exemplary implementations of the present invention willnow be descried in greater detail.

[0149] To describe the encoding of SP frame S1 or S2 in a normalbitstream, S1 is used as an example.

[0150] Here, DCT transform is performed to the macroblock of theoriginal video, and the obtained coefficients as K_(orig1) denoted

[0151] Next, after motion compensation, DCT transform is performed tothe predicted macroblock, and the obtained coefficients as K_(pred1)denoted.

[0152] Then the process includes subtracting K_(pred1) from K_(orig1)and obtaining error coefficients K_(err1).

[0153] K_(err1)=K_(orig1)−K_(pred1).

[0154] Next, K_(err1) is quantized using QP₁ and error levels L_(err1)obtained,

[0155] L_(err1)=QP₁ (K_(err1)).

[0156] Then entropy encoding is performed on L_(err1) and bitstream S1obtained.

[0157] Using the S1 decoder described above, for example, the processincludes reconstructing levels L_(rec1) and the reference for the nextframe encoding. Note that here there is a quantizer Qs₁ and adequantizer Qs₁ ⁻¹ in the reconstruction loop.

[0158] The encoding of switching bitstream S12, for example, whenswitching from bitstream 1 to bitstream 2, is based on the encoding ofS1 and S2.

[0159] For example, prediction coefficients K_(pred1) are quantized inthe S1 encoder using quantizer Qs₂,

[0160] L_(pred12)=Qs₂(K_(pred1)).

[0161] Then the process includes subtracting L_(pred12) from thereconstructed level L_(rec2) in S2 encoder.

[0162] L_(err12)=L_(rec2)−L_(pred12).

[0163] Next entropy encoding of L_(err12) is performed and bitstream S12generated.

[0164] Reference is now made to FIG. 9, which is a block diagramdepicting a decoder 900 for S1 and S2, in accordance with certain otherimplementations of the present invention. Here, it is noted that thequantization Qs is operated on the reconstructed DCT reference ratherthan on the decoded DCT residue and the DCT prediction. The quantizationin this example can be described as:

Y=[X*A(Qs)+2¹⁹]/2²⁰,

[0165] where X is the reconstructed DCT coefficient, and Y is thequantized DCT coefficient. A(.) is the quantization table. Qs is thequantization step.

[0166] If merging the dequantization QP and quantization QS in one step,the operation can, for example, be formularized as${L_{rec} = \frac{{\lbrack {{K_{pred}( {i,j} )} + {{L_{err}( {i,j} )}*\frac{( {2^{20} + {{A({Qp})}/2}} )}{A({Qp})}}} \rbrack*{A({Qs})}} + 2^{19}}{2^{20}}},$

[0167] where L_(err) are the levels of the prediction errorcoefficients, K_(pred) are prediction coefficients. This is quitedifferent from that used in conventional SP coding. One advantage isthat a high quality display can be reconstructed from the part in [. . .] of the above formula.

[0168] Therefore, decoder 900 provides two ways for reconstructing thedisplay image. In the first case, the reconstructed reference isdirectly used for the purpose of display. There is little if anycomplexity increase in this case. In the second case, if the decoder ispowerful enough, another high quality image can be reconstructed fordisplay. This process includes the modules within box 902. These modulesare, for example, non-normative parts for the current JVT standard.

[0169]FIG. 10 illustrates a decoder 1000 for the switching bitstreamS12, in accordance with certain further implementations of the presentinvention. In this example, decoder 1000 for the switching bitstream S12is slightly different from that for S1 and S2, presented in previoussections. Here, for example, the quantization Qs is only needed on theDCT prediction. Again, the quantization in this example can be describedas:

Y=[X*A(Qs)+2¹⁹]/2²⁰.

[0170] Decoder 1000 is configured to know which SP bitstream isreceived. Therefore, for example, a 1-bit syntax can be employed tonotify decoder 1000. An exemplary modification in the SP syntax andsemantic include a Switching Bitstream Flag (e.g., 1 bit) andQuantization parameter (e.g., 5 bits).

[0171] Thus, for example, when Ptype indicates an SP frame, the 1-bitsyntax element “Switching Bitstream Flag” is inserted before the syntaxelement “Slice Qp”. Here, when the Switching Bitstream Flag is 1, thecurrent bitstream is decoded as Bitstream S12, and the syntax element“Slice QP” is skipped; otherwise it is decoded as Bitstream S1 or S2,and the syntax element “Slice QP” is the quantization parameter Qp.

[0172] When Ptype indicates a SP frame, the syntax element “SP Slice QP”is inserted after the syntax element “Slice QP” to encode thequantization parameter Qs.

[0173] An encoder 1100 is illustrated in the block diagram of FIG. 11,in accordance with certain further exemplary implementations of thepresent invention.

[0174] Here, for example, encoder 1100 includes a switch 1102. Thus, theDCT prediction can be directly subtracted from the original DCT imagewithout quantization and dequantization, or the DCT prediction can besubtracted from the original DCT image after quantization anddequantization. Whether the DCT prediction is quantized or not can bedecided, for example, one by one coefficient with rate-distortioncriterion.

[0175] Thus, several exemplary improved SP picture coding methods andapparatuses have been presented. Separate Qs can be provided forup-switching and down-switching bitstreams. The Qs for switchingbitstream coding can be decoupled from the prediction and reconstructionloop. This eliminates the contradiction of reducing switching bitstreamsize and improving coding efficiency of the normal bitstreams, forexample. There can also be significant reduction in the switchingbitstream size while maintaining the high coding efficiency of thenormal bitstreams by optimizing different Qs independently. SomeQuantization/Dequantization processes can be removed in accordance withcertain implementations to improve coding efficiency. Coding efficientlycan also be improved by using the same reference for prediction andreconstruction. The methods and apparatuses may also be configured toallow for more down-switching points than up-switching points.

[0176] Conclusion:

[0177] Although the description above uses language that is specific tostructural features and/or methodological acts, it is to be understoodthat the invention defined in the appended claims is not limited to thespecific features or acts described. Rather, the specific features andacts are disclosed as exemplary forms of implementing the invention.

What is claimed is:
 1. An encoding method comprising: encoding data intoa first bitstream using a first quantization parameter; encoding saiddata into a second bitstream using a second quantization parameter thatis different from said first quantization parameter; and generating anencoded switching bitstream associated with said first and secondbitstreams using said first quantization parameter to supportup-switching between said first and second bitstreams and using saidsecond quantization parameter to support down-switching between saidfirst and second bitstreams.
 2. The method as recited in claim 1,wherein said first quantization parameter and said second quantizationparameter are decoupled.
 3. The method as recited in claim 1, whereinencoded switching bitstream is configured to support a plurality ofup-switching periods and a plurality of down-switching periods.
 4. Themethod as recited in claim 3, wherein over a period of time a number ofsaid down-switching periods is greater than a number of saidup-switching periods.
 5. The method as recited in claim 1, wherein saidfirst bitstream and said second bitstream have different data bit rates.6. A computer-readable medium comprising computer-implementableinstructions for causing at least one processing unit to perform actscomprising: encoding data into a first bitstream using a firstquantization parameter; encoding said data into a second bitstream usinga second quantization parameter that is different from said firstquantization parameter; and generating an encoded switching bitstreamassociated with said first and second bitstreams using said firstquantization parameter to support up-switching between said first andsecond bitstreams and using said second quantization parameter tosupport down-switching between said first and second bitstreams.
 7. Thecomputer-readable medium as recited in claim 6, wherein said firstquantization parameter and said second quantization parameter aredecoupled.
 8. The method as recited in claim 6, wherein encodedswitching bitstream is configured to support a plurality of up-switchingperiods and a plurality of down-switching periods.
 9. Thecomputer-readable medium as recited in claim 8, wherein over a period oftime a number of said down-switching periods is greater than a number ofsaid up-switching periods.
 10. The computer-readable medium as recitedin claim 6, wherein said first bitstream and said second bitstream havedifferent data bit rates.
 11. An apparatus comprising: a first bitstreamencoder configured to encode data into an encoded first bitstream usinga first quantization parameter; a second bitstream encoder configured toencode said data into an encoded second bitstream using a secondquantization parameter that is different from said first quantizationparameter; and a switching bitstream encoder operatively coupled to saidfirst bitstream encoder and said second bitstream encoder and configuredto output an encoded switching bitstream that supports up-switching anddown-switching between said first encoded bitstream and said secondencoded bitstream based on information processed using said first andsecond quantization parameters.
 12. The apparatus as recited in claim11, wherein said first quantization parameter and said secondquantization parameter are decoupled.
 13. The apparatus as recited inclaim 11, wherein said encoded switching bitstream is configured tosupport a plurality of up-switching periods and a plurality ofdown-switching periods.
 14. The apparatus as recited in claim 13,wherein over a period of time a number of said down-switching periods isgreater than a number of said up-switching periods.
 15. The apparatus asrecited in claim 11, wherein said first bitstream and said secondbitstream have different data bit rates.
 16. A decoding methodcomprising: receiving at least one encoded bitstream selected from agroup comprising a first bitstream that was generated using a firstquantization parameter and a second bitstream that was generated using asecond quantization parameter that is different from said firstquantization parameter; decoding said received encoded bitstream;receiving an encoded switching bitstream associated with said first andsecond bitstreams that was generated using said first quantizationparameter to support up-switching between said first and secondbitstreams and using said second quantization parameter to supportdown-switching between said first and second bitstreams; and decodingsaid received encoded switching bitstream using said first and secondquantization parameters.
 17. The method as recited in claim 16, whereinsaid first quantization parameter and said second quantization parameterare decoupled.
 18. The method as recited in claim 16, wherein decodingsaid received encoded switching bitstream occurs during at least oneperiod selected from a group comprising at least one of a plurality ofup-switching periods and at least one of a plurality of down-switchingperiods.
 19. The method as recited in claim 18, wherein over a period oftime a number of said down-switching periods is greater than a number ofsaid up-switching periods.
 20. The method as recited in claim 16,wherein said first bitstream and said second bitstream have differentdata bit rates.
 21. A computer-readable medium comprisingcomputer-implementable instructions for causing at least one processingunit to perform acts comprising: receiving at least one encodedbitstream selected from a group comprising a first bitstream that wasgenerated using a first quantization parameter and a second bitstreamthat was generated using a second quantization parameter that isdifferent from said first quantization parameter; decoding said receivedencoded bitstream; receiving an encoded switching bitstream associatedwith said first and second bitstreams that was generated using saidfirst quantization parameter to support up-switching between said firstand second bitstreams and using said second quantization parameter tosupport down-switching between said first and second bitstreams; anddecoding said received encoded switching bitstream using said first andsecond quantization parameters.
 22. The computer-readable medium asrecited in claim 21, wherein said first quantization parameter and saidsecond quantization parameter are decoupled.
 23. The computer-readablemedium as recited in claim 21, wherein decoding said received encodedswitching bitstream occurs during at least one period selected from agroup comprising at least one of a plurality of up-switching periods andat least one of a plurality of down-switching periods.
 24. Thecomputer-readable medium as recited in claim 23, wherein over a periodof time a number of said down-switching periods is greater than a numberof said up-switching periods.
 25. The computer-readable medium asrecited in claim 21, wherein said first bitstream and said secondbitstream have different data bit rates.
 26. An apparatus comprising: afirst decoder configured to decode a first encoded bitstream into adecoded first bitstream using a first quantization parameter; a seconddecoder configured to decode a second bitstream into a decoded secondbitstream using a second quantization parameter that is different fromsaid first quantization parameter; and a switching bitstream decoderoperatively coupled to said first decoder and said second decoder andconfigured to output a decoded switching bitstream that supportsup-switching and down-switching between said first decoded bitstream andsaid second decoded bitstream based on information processed using saidfirst and second quantization parameters.
 27. The apparatus as recitedin claim 26, wherein said first quantization parameter and said secondquantization parameter are decoupled.
 28. The apparatus as recited inclaim 26, wherein decoding said received encoded switching bitstreamoccurs during at least one period selected from a group comprising atleast one of a plurality of up-switching periods and at least one of aplurality of down-switching periods.
 29. The apparatus as recited inclaim 28, wherein over a period of time a number of said down-switchingperiods is greater than a number of said up-switching periods.
 30. Theapparatus as recited in claim 26, wherein said first bitstream and saidsecond bitstream have different data bit rates.
 31. A decoding methodcomprising: reconstructing DCT reference data; and quantizing saidreconstructed DCT reference data using a quantization step (Qs) on thereconstructed DCT reference and not on decoded DCT residue and the DCTprediction data.
 32. The decoding method as recited in claim 31, whereinquantizing said reconstructed DCT reference data is represented by:Y=[X*A(Qs)+2¹⁹]/2²⁰, wherein X includes a reconstructed DCT coefficientand Y includes a quantized DCT coefficient, and A(.) is associated witha quantization table
 33. The decoding method as recited in claim 31,further comprising: merging a dequantization QP and quantization QS into one operation represented by:${L_{rec} = \frac{{\lbrack {{K_{pred}( {i,j} )} + {{L_{err}( {i,j} )}*\frac{( {2^{20} + {{A({Qp})}/2}} )}{A({Qp})}}} \rbrack*{A({Qs})}} + 2^{19}}{2^{20}}},$

where L_(err) are levels of the prediction error coefficients, K_(pred)are prediction coefficients.
 34. The decoding method as recited in claim33, further comprising generating data for a high quality display basedat least in part on${K_{pred}( {i,j} )} + {{L_{err}( {i,j} )}*{\frac{( {2^{20} + {{A({Qp})}/2}} )}{A({Qp})}.}}$


35. The decoding method as recited in claim 31 further comprising:selectively reconstructing different qualities of display images. 36.The decoding method as recited in claim 31 further comprising receivingdecoder notification data associated with at least one switching event.37. The decoding method as recited in claim 36, wherein said decodernotification includes at least a one-bit syntax having a switchingbitstream flag.
 38. The decoding method as recited in claim 37, whereinsaid decoder notification data includes at least one quantizationparameter.
 39. A computer-readable medium having computer-implementableinstructions for causing at least one processing unit to perform actscomprising: decoding bitstream data by: reconstructing DCT referencedata; and quantizing said reconstructed DCT reference data using aquantization step (Qs) on the reconstructed DCT reference and not ondecoded DCT residue and the DCT prediction data.
 40. Thecomputer-readable medium as recited in claim 39, wherein quantizing saidreconstructed DCT reference data is represented by: Y=[X*A(Qs)+2¹⁹]/2²⁰,wherein X includes a reconstructed DCT coefficient and Y includes aquantized DCT coefficient, and A(.) is associated with a quantizationtable
 41. The computer-readable medium as recited in claim 39, havingcomputer-implementable instructions for causing the at least oneprocessing unit to perform further acts comprising: merging adequantization QP and quantization QS in to one operation representedby:${L_{rec} = \frac{{\lbrack {{K_{pred}( {i,j} )} + {{L_{err}( {i,j} )}*\frac{( {2^{20} + {{A({Qp})}/2}} )}{A({Qp})}}} \rbrack*{A({Qs})}} + 2^{19}}{2^{20}}},$

where L_(err) are levels of the prediction error coefficients, K_(pred)are prediction coefficients.
 42. The computer-readable medium as recitedin claim 41, having computer-implementable instructions for causing theat least one processing unit to perform further acts comprising:comprising generating data for a high quality display based at least inpart on${K_{pred}( {i,j} )} + {{L_{err}( {i,j} )}*{\frac{( {2^{20} + {{A({Qp})}/2}} )}{A({Qp})}.}}$


43. The computer-readable medium as recited in claim 39 havingcomputer-implementable instructions for causing the at least oneprocessing unit to perform further acts comprising: selectivelyreconstructing different qualities of display images.
 44. Thecomputer-readable medium as recited in claim 39 havingcomputer-implementable instructions for causing the at least oneprocessing unit to perform further acts comprising: receiving decodernotification data associated with at least one switching event.
 45. Thecomputer-readable medium as recited in claim 44, wherein said decodernotification includes at least a one-bit syntax having a switchingbitstream flag.
 46. The computer-readable medium as recited in claim 45,wherein said decoder notification data includes at least onequantization parameter.
 47. A decoder comprising: logic operativelyconfigured to reconstruct DCT reference data and quantize saidreconstructed DCT reference data using a quantization step (Qs) on thereconstructed DCT reference and not on decoded DCT residue and the DCTprediction data.
 48. The decoder as recited in claim 47, wherein saidlogic quantizes said reconstructed DCT reference data as:Y=[X*A(Qs)+2¹⁹]/2²⁰, wherein X includes a reconstructed DCT coefficientand Y includes a quantized DCT coefficient, and A(.) is associated witha quantization table
 49. The decoder as recited in claim 47, whereinsaid logic merges a dequantization QP and quantization QS into oneoperation represented by:${L_{rec} = \frac{{\lbrack {{K_{pred}( {i,j} )} + {{L_{err}( {i,j} )}*\frac{( {2^{20} + {{A({Qp})}/2}} )}{A({Qp})}}} \rbrack*{A({Qs})}} + 2^{19}}{2^{20}}},$

where L_(err) are levels of the prediction error coefficients, K_(pred)are prediction coefficients.
 50. The decoder as recited in claim 49,wherein said logic is further configured to generate data for a highquality display based at least in part on${K_{pred}( {i,j} )} + {{L_{err}( {i,j} )}*{\frac{( {2^{20} + {{A({Qp})}/2}} )}{A({Qp})}.}}$


51. The decoder as recited in claim 47, wherein said logic si fitehrconfigured to selectively reconstruct different qualities of displayimages.
 52. The decoder as recited in claim 47 wherein said logic isfurther configured to receive decoder notification data associated withat least one switching event.
 53. The decoder as recited in claim 52,wherein said decoder notification includes at least a one-bit syntaxhaving a switching bitstream flag.
 54. The decoder as recited in claim53, wherein said decoder notification data includes at least onequantization parameter.